人类想要记住什么？

What Do Humans Want to Remember?

进食是人类生存的本质之一。进食，往坏了说是必需，往好了说是愉悦。几千年来，人类通过符号和语言传递着信息。这些信息以口耳相传的方式流传了一段时间，最终被记录下来。如今，我们拥有数百至数千年前关于各种事物的文字记录，但为了简洁起见，我们先来谈谈烹饪^。1

An essential aspect of what it is to be human is to eat. Eating is an act of necessity at worst and an act of pleasure at best. Humans, as we would have it, have passed down information for thousands of years through the use of symbols and language. This information persisted orally for some time and ultimately was written down. We now have written records communicating information from hundreds to thousands of years ago about all kinds of things, but for brevity, let’s focus on cooking.¹

在这种情况下，我们设想自己处于最基本的形态：我们记忆信息的能力有限，缺乏经验，也无法将外部信息嵌入我们的知识库。在这种有限的状态下，我们必须找到一种积累经验并将其运用的方法，以实现我们的目标，并提升我们更好地实现这一目标的能力。为了探索在这种有限状态下学习的方法，我们将遵循一个秘诀。重申一下，在我们当前抽象的形态下，我们的记忆力有限，嵌入额外信息的能力也有限（我们只知道我们能读懂的东西），我们需要积累经验。以下是我们的秘诀和配料表：

In this scenario, we imagine ourselves in our most basic form: we have limited capacity to hold information in memory, we lack experience, and we are unable to embed outside information into our knowledge base. In this limited form, we must find a way of gaining experience and carrying it forward to accomplish our goal, and to increase our ability to accomplish this goal well. To explore the methods of learning in this limited state, we’ll follow a recipe. To reiterate, in our current, abstracted form, we have limited memory and limited ability to embed additional information (we know only what we can read), and we need to gain experience. Here is our recipe and ingredient list:

• 原料：
• Ingredients:
• 1杯白糖
• 1 cup white sugar
• 半杯黄油
• ½ cup butter
• 2个鸡蛋
• 2 eggs
• 2茶匙香草精
• 2 teaspoons vanilla extract
• 1.5 杯通用面粉
• 1 ½ cups all-purpose flour
• 1 ¾ 茶匙泡打粉
• 1 ¾ teaspoons baking powder
• 半杯牛奶
• ½ cup milk

指示：

Instructions:

0. 抓取食材^。2

1. Grab ingredients.²

2. 将烤箱预热至 350°F (175°C)。在 9×9 英寸的烤盘上涂抹油脂和面粉，或在松饼烤盘上铺上纸垫。

3. Preheat the oven to 350°F (175°C). Grease and flour a 9×9-inch pan or line a muffin pan with paper liners.

4. 在中等大小的碗中，将糖和黄油搅拌在一起。

5. In a medium bowl, cream together the sugar and butter.

6. 逐个打入鸡蛋，然后加入香草搅拌。

7. Beat in the eggs, one at a time; then stir in the vanilla.

8. 将面粉和发酵粉混合，加入奶油混合物中，搅拌均匀。

9. Combine flour and baking powder, add it to the creamed mixture, and mix well.

10. 最后，加入牛奶搅拌至面糊顺滑。将面糊倒入或用勺子舀入准备好的锅中。

11. Finally, stir in the milk until the batter is smooth. Pour or spoon the batter into the prepared pan.

12. 在预热好的烤箱中烘烤30至40分钟。纸杯蛋糕烘烤20至25分钟。蛋糕摸起来有弹性就表示烤好了。

13. Bake for 30 to 40 minutes in the preheated oven. For cupcakes, bake for 20 to 25 minutes. Cake is done when it springs back to the touch.

尽管我们还不知道，但我们最终会进行多次尝试，因此我们将从蛋糕 0 开始，用数字来指代每一个。

Although we don’t know it yet, we will end up making multiple attempts, so we’ll refer to each one numerically, starting with Cake 0.

对于蛋糕 0，你开始阅读说明。你会看到总共七个按顺序排列的步骤。你从步骤 0 开始，因为它首先出现。你拿起所有原料，阅读说明，心想：“谁需要这些？反正它们最终都会放进碗里。” 所以，除了拿起原料（步骤 0）之外，你认为你可以按照最适合自己的顺序进行操作。

For Cake 0, you begin reading the instructions. You see a total of seven steps, in sequence. You start at step 0, as it appears first. You grab all the ingredients and read the directions and think, “Who needs these? It all ends up in the bowl anyway.” So besides grabbing the ingredients (step 0), you think that you can just do what you want in the order that suits you best.

你觉得混合面粉和泡打粉（步骤4）听起来最简单。你把它们倒在一起，然后去拿打好的奶油混合物。令你惊讶的是，你似乎找不到它。于是你读了步骤4之前的步骤：“3. 逐个打入鸡蛋；然后加入香草精。” 你照做了，然后把它加到面粉和泡打粉里。你以为快完成了，这时你注意到还有一些未用到的材料——糖、黄油和牛奶。你决定一次性把它们全部加进去搅拌。

You think that combining the flour and baking powder (step 4) sounds easiest. You dump them together and go to grab the creamed mixture. To your surprise, you cannot seem to find it. So you read the step before step 4: “3. Beat in the eggs, one at a time; then stir in the vanilla.” You do this, and add it to your flour and baking powder. You think you’re about finished when you notice you have unused ingredients—namely, the sugar, butter, and milk. You decide to just add them all in at once and stir.

你最终的混合物看起来不错，准备放进烤箱。烤箱不热，你又看了一遍说明书，发现第一步提示预热烤箱。你怒不可遏，纳闷为什么说明书没有在第0步写“按步骤操作”！你等了几个小时才等到蛋糕烤好，结果糟透了。你告诉自己，好吧，下次我一定按步骤操作。

You have a final mixture that looks fine and go to put it in the oven. It is not hot, so you read the instructions again, seeing that step 1 tells you to preheat the oven. Furious, you wonder why the instructions didn’t say “Instructions are in order” as step 0! You wait a couple of hours for your cake to finally be done, and it sucks. You tell yourself, fine, next time I will follow the instructions in order.

对于蛋糕1，你决定按照明确的顺序进行操作。你按照步骤0、步骤1、步骤2和步骤3进行操作。最后你到了步骤4：“将面粉和泡打粉混合，加入打好的奶油混合物中，搅拌均匀。” 你四处寻找打好的奶油混合物，却仍然找不到。你又一次因为说明书撒了谎而怒不可遏，于是把面粉和泡打粉也加到了打好的奶油混合物中。反正都一样。

For Cake 1, you decide that you will follow the instructions in explicit order. You do step 0, step 1, step 2, and step 3. You get to step 4: “Combine flour and baking powder, add it to the creamed mixture, and mix well.” You look around for the creamed mixture and still cannot find it. Livid that the instructions lied again, you add the flour and baking powder to the beaten mixture. It’s all the same anyway.

你基于上一步的成功完成了第四步（没有用该死的奶油混合物），但到了第五步（“最后，加入牛奶搅拌至面糊顺滑。将面糊倒入或用勺子舀入准备好的烤盘。”），你却找不到“准备好的烤盘”，而“面糊”又是什么呢？你拿起“9×9英寸烤盘”，把所有搅拌好的东西都倒进碗里。你把它放进“预热”的烤箱里，30分钟后，一个美味的蛋糕就出炉了！

You complete step 4 based on your success from the previous step (without the darned creamed mixture), but at step 5 (“Finally, stir in the milk until the batter is smooth. Pour or spoon the batter into the prepared pan.”), you can’t find the “prepared pan,” and what the heck is the “batter”? You grab your “9×9–inch pan” and pour in everything you’ve been mixing in the bowl. You place it into your “preheated” oven, and 30 minutes later you get a delicious cake!

按照步骤操作，你成功地做出了蛋糕！你还得出结论，“打发的混合物”肯定是“打发的混合物”，“碗里的东西”是“面糊”，“9×9英寸的烤盘”和“准备好的烤盘”是一样的。既然你已经成功地做出了蛋糕，你觉得现在没有什么可学的了，所以你努力把所有需要知道的东西都记在脑子里。然而，一些错误会不时地出现在你的脑海里，提醒你你不知道这些东西叫什么，你真希望自己知道哪些东西真正重要，哪些不重要（谁会在乎到第5步它就叫“面糊”呢？）。所以，让我们学习如何忘记。

By following the steps in order, you successfully made the cake! You also conclude that the “beaten mixture” must be the “creamed mixture,” the “stuff in the bowl” is the “batter,” and the “9×9–inch pan” is the same as the “prepared pan.” You feel like there is nothing left to learn now that you can successfully make the cake, so you try to commit everything you need to know to memory. However, some mistakes keep popping up in your head every now and then, reminding you that you didn’t know what things were called, and you wish you knew what was actually important to remember and what wasn’t (who cares that by step 5 it’s called “batter”?). So let’s learn how to forget.

现在，我们来谈谈蛋糕2。你决定一周后再做一次，但不幸的是，除了配料表，你把所有东西都弄丢了。你试图回忆整个食谱，但只记得三件事：

Now, for Cake 2. You decide to make the cake again a week later but, unfortunately, you lost everything but the ingredient list. You try to recall the entire recipe but can remember only three things:

1. 该食谱的制作顺序是先将烤箱预热至 350°F，然后准备一个涂有油脂的平底锅。

2. There is an order to the recipe that begins with preheating the oven to 350°F and getting a greased pan.

3. 应先混合湿成分，然后再混合干成分，最后形成“面糊”。

4. The wet ingredients should be mixed before the dry ingredients, finally ending up with the “batter.”

5. 面糊放入烤箱，直到变成蛋糕。

6. The batter goes into the oven until it turns into a cake.

你准备好所有材料，将糖和黄油混合在一起，然后加入鸡蛋和香草精，再加入牛奶，最后加入面粉和泡打粉，调成“面糊”（理想情况下，牛奶应该最后加入，但这样做也不会造成太大问题）。你把面糊倒入之前抹油的烤盘，烘烤30分钟。打开烤箱，扑面而来的蛋糕香味扑鼻而来。

You grab your ingredients and proceed to add the sugar and butter together, then add the eggs and vanilla, then the milk, and finally the flour and baking powder, producing the “batter” (you ideally should have added the milk last, but this does not cause undue problems). You add the batter to the pan you greased at the beginning, and bake it for 30 minutes. Upon opening the oven, you are greeted by the delicious smell of your cake.

你不再受说明书的束缚！你只需要记住几个关键步骤的顺序、各种原料的组合，以及将它们混合在一起，就能得到制作蛋糕的面糊！你意识到你不需要记住整个食谱；你可以将基本原理与原料清单结合起来，做出一个蛋糕！凭借你新学到的烘焙技巧，你就能鱼与熊掌兼得。

No longer are you limited by the instructions! All you had to remember was the order of a few key aspects, the combination of ingredients, and that by mixing them together, you get batter that makes a cake! You realize that you didn’t need to remember the entire recipe; you could use an underlying pattern in combination with the ingredient list to make a cake! With your newfound baking skills, you can have your cake and eat it too.

用于语言建模的神经网络

Neural Networks for Language Modeling

让我们使用美国内政部“关于蝙蝠的 13 个惊人事实”中的这句话作为示例，其中详细介绍了各种大小的蝙蝠：

Let’s use this sentence from the US Department of the Interior’s “13 Awesome Facts About Bats” detailing various sizes of bats as our running example:

蝙蝠的体型大小不一，从体重不到一便士的基蒂猪鼻蝠（也称为熊蜂蝠，是世界上最小的哺乳动物）到翼展可达 6 英尺的狐蝠。

Bats range in size from the Kitti’s hog-nosed bat (also called the bumblebee bat) that weighs less than a penny—making it the world’s smallest mammal—to the flying foxes, which can have a wingspan of up to 6 feet.

想象一下，你读了很多关于这种有趣哺乳动物的资料，现在准备玩一个预测下一个单词的游戏。给出“蝙蝠的活动范围”的提示，你会想到几个合理的单词——“长度”、“重量”、“大小”、“颜色”等等——然后你会从中选择一个。本章的剩余部分将讨论神经网络如何模拟这种结果，并选择一个单词来跟随给定的序列。

Imagine that you have done a lot of reading on this interesting mammal and are going to play the game of predicting the next word. Given the prompt of “Bats range in,” you will think of several reasonable words to follow—“length,” “weight,” “size,” “color,” and so on—and you will choose one of them. We will spend the rest of this chapter discussing how a neural network would mimic this outcome and choose a word to follow a given sequence.

作为参考，回想一下概率语言模型，它在 “语言模型如何表示语言？”中有简要描述。三元模型会为可能跟在短语“Bats range in”（以及其他三元模型）后面的每个单词的非零概率创建一个表示，并输出最可能的单词，如图2-1所示。

As a reference, recall the mention of probabilistic language models, briefly described in “How Does an LM Represent Language?”. A trigram model would create a representation for the nonzero probabilities of every word that might follow the phrase “Bats range in” (as well as other trigrams) and output the most likely one, as shown in Figure 2-1.

图 2-1：运行经过训练的概率三元语言模型。

那么，神经网络如何预测下一个单词？

So, how does a neural network predict the next word?

用神经网络预测下一个单词

Predicting the Next Word with Neural Networks

我们不会回过头去介绍人工神经网络，讨论它们的总体结构是如何受到观察到的生物神经元行为的启发等等。让我们直接讨论一个简单神经网络的基本结构，如图2-2所示。

We are not going to walk back all the way to introducing artificial neural networks, discussing how their general structure was inspired by the observed behavior of biological neurons, and so on. Let’s jump straight to discussing the basic structure of a simple neural network, as shown in Figure 2-2.

图 2-2。人工神经网络。

需要明确的是，神经网络并不等同于语言模型 (LM)。我们讨论的是以神经网络形式构建的语言模型，但神经网络可以执行许多其他任务，例如图像处理领域。人工神经网络包括以下内容：

To be clear, a neural network is not equivalent to an LM. We are discussing language models that are constructed as neural networks—but neural networks can work on many other tasks, such as in the area of image processing. An artificial neural network includes the following:

输入层

Input layer

使用适当结构化的输入集。

Consumes an appropriately structured set of inputs.

隐藏层（一个或多个）

Hidden layer (one or more)

对输入应用一组转换。

Applies a set of transformations to the inputs.

输出层

Output layer

有时也称为激活层，它将前面转换的结果转换为产生最终输出的东西。

Sometimes also referred to as the activation layer, this converts the result of the preceding transformations into something for producing the final output.

输入层将每个单词（或标记）转换为神经网络其余部分可以理解的向量表示——也就是我们在上一章讨论过的词嵌入！人们通常会在这里使用一组现有的预训练词嵌入，例如Word2Vec、GloVe或ELMo。

The input layer converts each word (or token) into a vector representation that the rest of the neural network can understand—namely, the word embedding we discussed in the previous chapter! Frequently, people will use an existing set of pretrained word embeddings here, such as Word2Vec, GloVe, or ELMo.

在本章中，我们将通过仅考虑一个隐藏层来简化问题。实际上，深度神经网络之所以被称为深度神经网络，是因为它们堆叠了多个隐藏层，并且每层学习到不同的权重。在此步骤中，网络将一组权重应用于嵌入。这些权重的值是模型的参数。该模型的训练方法是： 输入一段文本（“蝙蝠的范围”），将其生成的输出与文本中的真实值（“大小”）进行比较，并根据接近程度调整其隐藏层的权重。此过程重复多次，每次都会略微改变隐藏层的状态。4^希望您现在对这样一个模型需要处理多少文本以及需要调整多少参数才能构建一个质量不错的表示有了一定的了解！

We will simplify matters in this chapter by considering only one hidden layer. In reality, deep neural networks are so called because they stack multiple hidden layers, with different weights learned in each layer. In this step, the network applies a set of weights to the embeddings. The values of these weights are the parameters of the model. This model is trained by consuming a snippet of text as input (“Bats range in”), comparing its generated output to the true value in the text (“size”), and adjusting the weights in its hidden layer depending on how close it got. This process is repeated many times over, slightly altering the state of the hidden layer each time.⁴ Hopefully, you are now gaining some intuition about just how much text such a model needs to consume, and how many parameters it needs to tweak, in order to construct a decent-quality representation!

在隐藏层应用转换步骤后，神经网络已完成大部分工作。但隐藏层中的信息无法以其当前形式直接用于任何用途。输出层会问“那又怎样？”这个问题，它使用激活函数（再次借用了生物学术语）来实现这一点，激活函数的作用是将矩阵中的信息聚合成某种可传输的输出。确切地说，激活函数也是深度神经网络中一个隐藏层将信息传递到下一个隐藏层的机制。为简单起见，我们将绕过这部分的讨论，仅关注输出层的激活函数^。5

Having applied the transformation steps in its hidden layer, the neural network has done most of its work. But the information cannot be directly used for anything in its current form in the hidden layer. The output layer asks the question of “so what?” It does so using an activation function—again, terminology borrowed from biology—whose job is to aggregate the information in the matrices into some kind of transmissible output. To be precise, activation functions are also the mechanism by which one hidden layer passes information forward to the next in a deep neural network. For simplicity, we will bypass that part of the discussion and focus only on the activation function in the output layer.⁵

对于我们的任务，我们需要得到“Bats range in”后面所有可能出现的单词的概率分布，就像图 2-1中三元语言模型的概率表一样。概率分布是介于 0 到 1 之间的有限数字集合，其和为 1。因此，我们会应用特定的数学函数（例如 softmax）来生成概率 分布。

For our task, we need to end up with a probability distribution across all possible words that might follow “Bats range in,” just as we have in the probabilities table for the trigram language model in Figure 2-1. A probability distribution is a finite set of numbers between 0 and 1 that sum to 1. Therefore, a particular mathematical function is applied (e.g., softmax) to yield the probability distribution.

人工神经网络的“内部”是什么？是矩阵。图 2-3展示了一个简化的示例。

What is “inside” an artificial neural network? Matrices. Figure 2-3 shows a simplified example.

图 2-3。神经网络各层的组成。实际数字毫无意义，并且为了便于可视化，矩阵大小被显著截断。

激活层和聚合层中的值与前面所示的三元语言模型（图 2-1 ）的概率表中的值相当。对矩阵执行的操作由箭头表示。因此，例如，当我们谈到对嵌入应用权重时，这实际上是对这两层中的矩阵执行的操作的简写。

The values in the activation and aggregation layer are comparable to the values in the probabilities table of the trigram language model (Figure 2-1) shown earlier. The operations performed on the matrices are implied by the arrows. Thus, for instance, when we speak of weights being applied to the embeddings, this is shorthand for an operation being performed on the matrices in those two layers.

笔记

神经网络由什么组成？矩阵，以及在矩阵上执行的运算。

What is a neural network made of? Matrices, and operations defined to be performed on the matrices.

重要的是要理解，这种神经架构不仅限于创建语言模型（LM，一种用于预测下一个单词的系统）。这种神经架构可以承担的其他任务包括词性标注、命名实体识别、句子分类（例如，用于情感分析）等等，更不用说各种各样的图像识别任务了！将该架构应用于这些任务意味着需要改变输出层的功能和预期输出（当然，对于图像任务，输入也需要改变）。我们不会在本报告（专门讨论语言建模）中探讨这些其他方面，但希望避免在这一点上造成任何潜在的混淆。

It is important to understand that this neural architecture is not limited to creating an LM—a system for predicting the next word. Other tasks that this neural architecture can take on include part-of-speech tagging, named entity recognition, sentence classification (e.g., for sentiment), and so on, not to mention the myriad of tasks for image recognition! Applying the architecture to such tasks means altering the functionality of the output layer and the expected output (and for image tasks, the input as well, of course). We are not going to explore these other aspects in this report (which is specifically about language modeling) but wanted to avoid any potential confusion on this point.

到目前为止，我们讨论的是一个前馈、固定宽度的神经网络（之所以这样描述，是因为它输入前一个输入来预测序列中下一个标记或单词，并且输入的大小是固定数量的标记）。回到我们的例句，神经网络会通过“滑动”的方式进行训练，将连续的三元组输入与一元组输出配对：

We have up to now discussed a feed-forward, fixed-width neural network (so described because it feeds in previous input to predict the next token, or word, forward in the sequence, and the size of the input is a fixed number of tokens). Returning to our example sentence, the neural network would train on it by “sliding” along, pairing consecutive trigram inputs and unigram outputs:

“蝙蝠种类繁多”，“大小”

“Bats range in,” “size”

“大小范围”，“从”

“range in size,” “from”

“尺寸从…开始”，“这”

“in size from,” “the”

等等。虽然这缓解了概率n -gram 模型的一些主要问题，例如稀疏性和存储问题，但使用前馈固定宽度的神经网络进行语言建模存在一些严重的缺陷。6出现位置相近的^单词会一起影响网络，而出现位置相距较远的单词之间的关系则无法被检测到。

And so on. Although this relieves some of the major problems with probabilistic n-gram models, such as those of sparsity and storage, using a feed-forward, fixed-width neural network for language modeling has some serious drawbacks.⁶ Words that appear close together will affect the network together, but relationships between words that appear further apart will not be detected.

在我们的例句中，由于中间插入了括号“（也称为大黄蜂蝙蝠）”，网络在处理“重量不到一便士”时会完全忘记“基蒂猪鼻蝙蝠”。换句话说，即使我们将固定宽度的窗口扩大到 5 个标记，模型也能从输入“重量不到一便士”中学习到完全相同的信息，无论它是在我们关于蝙蝠大小的句子中，还是在诸如“那整堆羽毛的重量仍然不到一便士”之类的句子中，尽管人类读者可能会以不同的方式思考这些东西的重量。围绕文本输入的上下文信息被神经网络完全忽略。

In our example sentence, the network will forget all about “the Kitti’s hog-nosed bat” by the time it reaches “weighs less than a penny” because of the intervening parenthetical “(also called the bumblebee bat).” In other words, even if we widened its fixed-width window to say, five tokens, the model would have learned exactly the same information from the input “weighs less than a penny” whether it was in our sentence about bat sizes or in a sentence such as “That whole pile of feathers still weighs less than a penny,” although the human reader will likely think about the weights of these things differently. The contextual information that surrounds the text input is completely ignored by the neural network.

想象一下，一只小型机器人蝙蝠配备了一个人工神经网络，用于学习（并在其中生存）其周围环境。使用这个神经网络就像赋予蝙蝠金鱼般的记忆力。每隔五秒钟，它就会彻底重新评估周围环境，决定在哪里寻找食物、如何躲避捕食者等等。它只会从这段时间内感知到的信息中获取线索，然后立即忘记一切（就像我们烘焙0号蛋糕时那样）。蝙蝠可能感觉饱了，但却不记得自己吃过东西；它可能飞得特别快，却不记得自己正在被主动追赶。我们可怜的蝙蝠如果能获取超越当前时刻的信息，将受益匪浅。

Imagine a little robot bat that has been outfitted with an artificial neural network for learning about (and surviving in) its environment. Using this neural network would be like giving our bat the proverbial memory of a goldfish. Every five seconds, it would completely reevaluate its environment and decide where to look for food, how to avoid predators, and so on. It would take cues only from what it could sense in that time period, and then immediately forget everything (as we did in the baking of Cake 0). The bat might feel full but have no memory of eating; it might be flying extra fast but not remember that it is being actively pursued. Our poor bat would really benefit from having access to information beyond the current moment.

因此，在本章的剩余部分，我们将讨论如何赋予神经网络（以及我们的机器人蝙蝠）更好的记忆力。我们保证接下来的讨论会进行得更快！

So in the rest of this chapter, we cover how to give neural networks (and our robot bat) a better memory. We promise the rest of the discussion will go more quickly!

RNN：将神经架构应用于语言建模

RNN: Adapting Neural Architecture to Language Modeling

在训练人工神经网络时，我们每次只会告诉它：“这是现在正在发生的事情”。如果我们可以在输入中添加“这是你刚刚学到的东西”，会怎么样？我们可以。这就是循环神经网络( RNN )，如图 2-4所示。

When training an artificial neural network, all we tell it each time is, “Here is what is happening now.” What if we could add, “and here is what you just learned” to the input? We can. Meet Recurrent Neural Networks (RNNs), illustrated in Figure 2-4.

与人工神经网络相比，循环神经网络 (RNN) 结构的主要区别在于，它在训练时会接受两个输入：当前标记，以及使用前一个标记后的隐藏层（大致是所有参数的值）。通过比较图 2-4和图 2-2，您可以看到 RNN 是如何利用输入序列（而非单个输入）进行训练的。

Compared to an artificial neural network, the main difference in the structure of an RNN is that it takes in two inputs as it trains: the current token, and the hidden layer (roughly, the values of all the parameters) after consuming the previous token. By comparing Figure 2-4 with Figure 2-2, you can see how RNNs are trained by using sequences of inputs, rather than single inputs.

图 2-4。在 RNN 中，隐藏层也将其先前的状态作为输入。

RNN 一次处理一个标记，因此我们称其为对序列的迭代，或时间步长。在我们的示例中，我们有三个时间步长。在时间步长 0，标记“bats”被传递到隐藏层，隐藏层会进行修改以反映输入的消耗。在下一次迭代（时间步长 1）中，标记“range”和上一时间步长结束时的隐藏层状态都作为输入传递。在时间步长 2，标记“in”和时间步长 1 结束时的隐藏层状态都作为输入传递。标记序列现已结束，时间步长 2 结束时的隐藏层状态通过输出层传递，如图 2-5所示。

The RNN processes one token at a time, and hence we speak of iterations over the sequence, or time steps. In our example, we have three time steps. At time step 0, the token “bats” is passed to the hidden layer, which is altered to reflect the consumption of the input. At the next iteration, time step 1, both the token “range” and the hidden layer’s state at the end of the previous time step are passed as input. At time step 2, both the token “in” and the hidden layer’s state at the end of time step 1 are passed as input. The token sequence has now ended, and the hidden layer’s state at the end of time step 2 is passed through the output layer, as shown in Figure 2-5.

图 2-5。信息在 RNN 中的流动，用阴影矩形表示。现在，隐藏层除了输入 token 之外，还将其先前的状态作为输入。

请注意，虽然我们目前只关注（并展示）最后一个时间步的输出，但每个时间步都已准备好输出下一个单词的最佳猜测。换个角度来看，RNN 产生的输出序列与输入序列的大小完全相同——我们只是忽略了它。这一事实将在下一章中变得重要。

Note that although we are currently interested in (and showing) the output of only the last time step, every time step is fully ready to output its best guess for the next word. Another way to look at it is that the RNN is producing an output sequence that is exactly the same size as the input sequence—we are just ignoring it. This fact will become important in the next chapter.

笔记

RNN 总是产生与输入序列大小完全相同的输出序列。

RNNs always produce an output sequence that is exactly the same size as the input sequence.

一个显而易见的方面（希望如此）是，RNN 可以处理任意长度的输入序列，而且无需指定窗口大小，这无疑是一个优势。理论上，RNN 可以无限传播其隐藏层信息！

One aspect that (hopefully) is immediately obvious is that RNNs can process input sequences of any length, and there is no need to specify a window size, which is certainly an advantage. In theory, an RNN could just keep propagating its hidden layer information forever!

啊，要是真这么简单就好了。实践证明，循环神经网络 (RNN) 无法成功获取太多步骤前的信息。为什么？因为无论模型在输入序列中走了多远，模型的大小（隐藏层的大小）都保持不变。在图 2-5中，表示隐藏层的阴影矩形大小没有增加，即使模型必须记住越来越多的文本信息。这意味着，回到蝙蝠的大小问题，如果我们允许序列与句子一样长，那么预测“蝙蝠的范围在”之后的下一个单词的结构就和预测“蝙蝠的大小范围……高达 6 英尺”（43 个标记）的结构一样。由于同样的原因，这种计算速度也会非常慢：在每个训练步骤中，网络不仅需要调整一次权重，还需要在“时间回溯”到序列的开头时反复调整权重^。7

Ah, if only it were that simple. In practice, it turns out that an RNN is not successful at accessing information from too many steps back. Why? Well, the size of the model (the size of the hidden layer) remains exactly the same, no matter how far along in the input sequence the model is. In Figure 2-5, the shaded rectangles representing the hidden layer are not growing in size, even though the model is having to remember information about an increasing amount of text. This means that, returning to our bat sizes, if we allow sequences to be as long as sentences, we are limited to the same structure for predicting the next word after “Bats range in,” as for “Bats range in size…up to 6 feet” (43 tokens). Such computation also turns out to be quite slow, for the same reason: at each training step, the network has to adjust its weights not just once, but over and over as it travels “back in time” to the beginning of the sequence.⁷

现在，我们已将蝙蝠模型升级为循环神经网络 (RNN)，并允许它记住更早一些的事情，例如它最近在进食，或者正在逃避追捕。但它仍然记不住三天前它去过一个理想的觅食地，也无法回头查看，因为这些信息已经被之后发生的各种细节所淹没。同样，它很容易因为一次糟糕的遭遇而受到惊吓，从此再也不会回到理想的觅食地。我们如何才能帮助我们的蝙蝠学会记住重要的事情，忘记不重要的事情呢？

We have now upgraded our bat to an RNN and allowed it to remember things a bit further back, such as that it was recently eating or in the middle of escaping pursuit. But it still can’t remember that it visited a promising feeding ground three nights ago, and to check back, because that information has been crowded out by all the minutiae of everything that came after. Similarly, it is liable to get spooked all out of proportion by a single bad encounter and never return to a good feeding spot again. How can we help our bat learn to remember the important things and forget the unimportant ones?

LSTM：为 RNN 添加单独的记忆结构

LSTM: Adding a Separate Memory Structure to RNN

长短期记忆( LSTM ) 网络以 RNN 架构为基础，并添加了一个持续存在的记忆状态（只要网络正在处理输入序列）。在每个时间步，隐藏层都会访问并修改记忆状态中的信息。图 2-6展示了 LSTM 网络的简化视图。

Long Short-Term Memory (LSTM) networks start with an RNN architecture and add a memory state that persists over time (for as long as the network is processing the input sequence). At each time step, the hidden layer both accesses and alters the information in the memory state. Figure 2-6 shows a simplified view of an LSTM network.

图 2-6。与 RNN（参见图 2-5 ）相比，LSTM 网络增加了随时间持续存在的记忆状态。

隐藏层和记忆状态之间的相互作用被称为门，因为它们充当信息的守门人，决定什么可以通过，什么不能通过。门与两个任务有关：确定从前一个时间步骤中忘记哪些信息，以及确定在未来记住哪些信息。如果没有遗忘能力，模型就会患上一种超忆症。8回想一下^前面关于语言模型学习内容的讨论：它力求在其“任务”中尽可能有效，即对于这样的输入，它应该产生那样的期望输出。模型中的信息并非都与特定任务相关，因此，这些信息除了占用空间和计算时间之外什么也不做。对于人类和语言模型来说，遗忘能力都是记忆的一个重要方面。

The interactions between the hidden layer and the memory state are called gates because they serve as gatekeepers of information, deciding what shall pass and what shall be denied passage. The gates are concerned with two tasks: determining which information to forget from the previous time steps and determining which information to remember going forward. Without the ability to forget, the model suffers from a sort of hyperthymesia.⁸ Recall the earlier discussion of what an LM learns: it seeks to be as effective as possible in its “task” that for input that is like this, it should produce desired output that is like that. Not all the information in the model is going to be relevant to the specific task, and therefore that information would be doing nothing but taking up space and computing time. The ability to forget is a significant aspect of memory, for both humans and LMs.

在每个时间步处理每个输入时，隐藏层仍在处理预测下一个单词的问题，而门则选择要保留的信息子集。无需赘述任何技术细节（这并非我们的目标），只需说明这些机制也是神经网络，并且以相同的方式进行训练即可。给定一个输入序列，LSTM 单元会同时学习如何预测下一个单词，以及如何记住序列中的重要部分并忽略不重要的部分。

As each input is processed at each time step, the hidden layer is still working on the predict-the-next-word problem, and the gates are choosing which subsets of information to retain. Without getting into any technical detail (which is not our goal), suffice it to say that these mechanisms are also neural networks, and they get trained in the same way. Given an input sequence, the LSTM unit simultaneously learns how to predict the next word and how to remember important parts of the sequence and forget unimportant ones.

笔记

对于人类和语言模型来说，遗忘能力都是记忆的一个重要方面。

The ability to forget is a significant aspect of memory, for both humans and LMs.

最后，我们的蝙蝠拥有一个长短期记忆（LSTM）。它可以记住重要的事情，例如特别肥沃的觅食地和躲避捕食者的好地方。蝙蝠也不会被琐事分心，例如它不能吃的植物的位置，或者群体中其他蝙蝠的具体行为。

Finally, our bat has an LSTM. It can remember important things, like a particularly fertile feeding grounds and good places to hide from predators. The bat will also not get distracted by trifles, such as the location of plants it can’t eat or the exact behavior of every other bat in its colony.

关于使用 RNN 的考虑

Considerations on the Use of RNNs

RNN，或者说一般意义上的神经网络，并不会将其应用范围限制在语言模型上。神经网络的开发不仅是为了语言建模，也是为了处理文本以外的各种数据，例如图像、音频等等。在本报告中，我们仅关注神经网络架构作为语言模型的应用，但这两者绝不等同。

Nothing about RNNs, or neural networks in general, constrains their applicability to only language models. Neural networks are developed for all sorts of tasks beyond modeling language, and all sorts of data beyond text, such as image, audio, and many others. In this report, we are concerned with the use of neural network architecture only as language models, but the two are in no way equivalent.

具有类似 RNN 架构的语言模型可用于各种自然语言处理 (NLP) 任务，例如分类和文本生成。我们之所以称之为类似 RNN 的架构，是因为我们仅使用 LSTM 来说明 RNN 基本架构的一种变体，而实践中存在多种变体。虽然它们的数学结构有所不同，但其主要原理是相同的：一个通过神经技术增强的 RNN，用于维持记忆状态。我们在此不再赘述，仅鼓励您注意变体的存在，并根据需要查找更多具体信息。

Language models with RNN-like architectures can be used for a variety of natural language processing (NLP) tasks, such as classification and text generation. We say RNN-like architectures, because we have used LSTMs to illustrate just one variation on the basic architecture of RNNs, and in practice multiple variations exist. While they differ in mathematical structure, the main intuition is the same: an RNN enhanced with neural techniques to maintain a memory state. We will not get into further detail here, and encourage you merely to be aware of the existence of variations and to seek out additional concrete information as needed.

此外，我们无需构建词级神经语言模型。我们可以轻松地构建（就像人类一样）字符级语言模型，它同样可以预测序列中的下一个字符。正如我们所提到的，这种神经架构可以承担的其他任务包括词性标注、命名实体识别、句子分类（例如，用于情感分析）、图像识别和时间序列预测。

Furthermore, nothing requires us to build word-level neural language models. We just as easily can build (as people have) character-level language models, which similarly predict the next character in a sequence. And as we’ve mentioned, other tasks that this neural architecture can take on include part-of-speech tagging, named entity recognition, sentence classification (e.g., for sentiment), image recognition, and time-series prediction.

最后，回想一下我们之前提到的关于拥有多个隐藏层的可能性，将它们堆叠起来，使得一个隐藏层也作为下一个隐藏层的输入（通过激活函数）。实践中，我们已经注意到堆叠几层（大约两到四层）可以提升性能，但堆叠太多层会引发一个我们之前讨论过的常见问题：信息开始消失。这应该会为本报告后面讨论 Transformer 架构时的一个主题提供一些提示：网络中现在有更多连接，而且是在层与层之间！

Finally, recall our earlier aside on the potential of having more than one hidden layer, stacking them so that one hidden layer also serves as the input to the next hidden layer (via an activation function). In practice, it has been noticed that stacking a few layers (somewhere in the two-to-four range) can improve performance, but stacking too many layers introduces a version of a familiar problem, as we’ve discussed: information starts to disappear. This should give a hint as to one topic to come later in the report, when we discuss Transformer architecture: more connections in the network, now between layers!

关键要点

Key Takeaways

本章介绍了如何使用神经网络构建语言模型，并重点介绍了两种用于文本处理的架构：RNN 和 LSTM。在继续讨论更复杂的架构之前，我们想先总结以下几个关键点：

This chapter has introduced how neural networks can be used to construct language models and then focused particularly on two architectures for text processing: RNNs and LSTMs. Before moving on to discuss more complex architectures, we would like to leave you with the following key takeaways:

• 为了应对日益复杂的语言建模前沿领域，对神经网络语言模型的基本构建模块建立良好的直觉至关重要。神经网络架构的简单理解可以分为三个主要部分：输入层（嵌入）、隐藏层（权重和记忆）和输出层（激活函数）。
• It is important to develop good intuition about the basic building blocks of a neural network language model, in order to tackle the increasingly complex models that are coming to the forefront of language modeling. A simple way to think about neural network architecture is in three main parts: the input layer (embeddings), the hidden layer (weights and memory), and the output layer (activation function).
• 对于语言模型 (LM) 来说，拥有顺序的概念非常重要，因为文本是一种序列数据，顺序至关重要。循环神经网络 (RNN) 是一类特别适合处理序列数据（例如文本）的神经网络。
• It is important for an LM to have a notion of order, because text is a type of sequence data, where order matters. RNNs are a class of neural networks particularly well suited to processing sequence data, such as text.
• 平衡的记忆方法至关重要，既要记住什么，也要忘记什么。LSTM 等架构相比 RNN 有了重要的改进，增加了记忆状态和信息门。
• A balanced approach to memory is important, both what to remember and what to forget. Architectures such as LSTMs offer an important improvement over RNNs, with the addition of a memory state and information gates.
• 对于可以访问完整输入任务的任务（例如翻译或摘要），使用双向模型通常是一个好主意。
• For tasks with access to the full input task, such as translation or summarization, it’s often a good idea to use a bidirectional model.
• 最后，正如语言模型 (LM) 不必构建为神经网络一样，循环神经网络 (RNN)（或其变体，例如长短期记忆 (LSTM)）也并非仅限于语言模型 (LM) 的功能。RNN 可以构建和训练用于许多其他任务，例如语音标注、命名实体识别、句子分类、图像处理等等。事实上，神经网络本身是一个比我们在本报告中涵盖的内容更为广泛的主题。
• Finally, just as an LM does not have to be constructed as a neural network, an RNN (or a variant thereof, such as an LSTM) is not limited to functioning only as an LM. An RNN can be constructed and trained for many other tasks, such as speech tagging, named entity recognition, sentence classification, image processing, and so on. Indeed, neural networks themselves are a more general topic than what we cover in this report.

人类如何总结？

How Do Humans Summarize?

我们首先来思考一下人类是如何完成文本摘要任务的。我们将用“你画我猜”游戏来更具体的例子。你们两人一组玩“你画我猜”游戏：你随机选择一个你的同伴没见过的单词，然后你必须画出这个单词的图形，让你的同伴猜对。图 3-1展示了游戏中单词“bat”步骤的一个简单示例。

Let’s begin by considering how humans approach the task of summarizing text. We will make the example more concrete by using the game of Pictionary. You play Pictionary in a team of two: you randomly select a word that your partner does not see, and you must then draw a picture of the word to enable your partner to guess the word correctly. Figure 3-1 shows a simple example of the steps in the game with the word “bat.”

常规的“你画我猜”游戏有时间限制：你的队伍有有限的时间进行绘画和猜测，如果你的同伴猜错，你的队伍就不会得分。但让我们稍微修改一下游戏，创建一个“你画我猜”的总结版。

Regular Pictionary has a time component: your team has a limited time for drawing and guessing, and if your partner fails to guess correctly, you do not pick up a point for your team. But let’s amend the game slightly and create a Summarization version of Pictionary.

图 3-1。简单的 Pictionary 示例（来源：美国内政部）。

现在，给你一段较长的文本（而不是一个单词），你必须将其转换成一幅图画，然后你的队友必须根据这段文字写出一个简短的总结句子——比如说，不超过10个单词。你会在你的图画里写些什么呢？我们来看图3-2中的例子。

Now, you are given a longer chunk of text (instead of a word), which you must transform into a drawing, and your teammate must then generate a short summary sentence from it—say, no more than 10 words long. What would you put in your drawing? Let’s take the example shown in Figure 3-2.

图 3-2。总结 Pictionary 示例（来源：美国内政部）。

逐步思考你自己的思维过程。记住你的任务：将一张图片解读成一个10字的摘要。你需要尽可能多地向负责撰写摘要的队友传达信息，同时确保队友了解最重要的信息（当然，这对你和队友来说都是主观的）。

Consider your own thought process step by step. You keep the task in mind: interpreting an image into a 10-word summary. You want to communicate as much information as possible to your teammate, who will create the summary, while also making sure your teammate is aware of the most important information (this will, of course, be subjective on both your part and theirs).

在阅读和绘画的过程中，你会判断哪些信息重要且必须传达：必须有一只蝙蝠和某种沙漠植物，并且蝙蝠看起来应该像正在以植物的花朵为食，以暗示其授粉和吸食花蜜。另一方面，除非你和你的队友是世界级的蝙蝠专家，否则你会决定忽略它是墨西哥长舌蝙蝠​​的事实，这并非因为你们不知道如何具体地画出这个物种。你决定不把这些信息写进你为队友画的画里。总而言之，你正在阅读文本，指出哪些部分很重要，哪些部分可以忽略，然后完成你的画作。

As you read and draw, you make judgment calls about which information is important and definitely needs to be passed along: there must be a bat, and some sort of desert plant, and the bat should look like it is feeding on the flowers of the plant to suggest both pollination and eating nectar. On the other hand, unless you and your teammate are world experts on bats, you decide to ignore the fact that it’s a Mexican long-tongued bat, not in the least because you have no idea how to specifically draw that species. That is information that you decide will not make it into your drawing for your teammate. To summarize, you are reading the text, noting some portions as important and others as to be ignored, and finalizing your drawing.

用于语言建模的编码器-解码器

Encoder-Decoders for Language Modeling

前面那句话本意是想引导一下，希望能让你想起 RNN 或 LSTM 架构！LSTM 也接受文本序列，选择性地记住和忘记，并细化隐藏表征（比如你的绘画）。但请记住，在每个时间步，LSTM 都会从左到右处理文本，并输出对下一个单词的最佳猜测。因此，在某种程度上，它可以为给定的文本输入序列生成文本输出序列，但存在两个问题：

That preceding sentence was meant to be a bit leading, and, hopefully, to remind you of an RNN or LSTM architecture! An LSTM also takes in a sequence of text, selectively remembers and forgets, and refines a hidden representation (like your drawing). But remember that at each time step, the LSTM is processing the text from left to right and outputs its best guess for the next word. So in some way, it can produce an output sequence of text for a given input sequence of text, but with two problems:

• 文本的输出序列必然与输入的大小完全相同。
• The output sequence of text is necessarily exactly the same size as the input.
• 在每个时间步，LSTM 都仅根据之前遇到的文本对下一个单词做出最佳猜测。相反，我们希望先处理整个文本，然后才能构建最佳摘要。
• At every time step, the LSTM is making its best guess for the next word based only on text it had previously encountered. Instead, we want the entire text to be processed first, and only then should the best summary be constructed.

第二个问题可以通过使用双向架构来解决，正如上一章所述。但这并不能缓解第一个问题：对于摘要任务（或许多其他任务，例如翻译）来说，将输出序列的大小与输入序列的大小绑定是没有意义的。

The second problem can be addressed by using a bidirectional architecture, as mentioned in the previous chapter. But this in no way alleviates the first problem: tying the size of the output sequence to that of the input sequence does not make sense for a summarization task (or plenty of other tasks, such as translation).

你的同伴是如何创作出所需的总结句子的？他们观察了你的画作，并生成了一个符合语法的句子，其中包含了他们认为你选择表达的显著信息。就像一个人可以将另一个人的画作转换成文本一样，当一个 LSTM 创建了“画作”后，我们可以用第二个 LSTM 将其转换回文本！

How did your partner create the required summary sentence? They looked at your drawing and generated a grammatical sentence that contained the salient information they believed you had chosen to represent. Just as one human can transform another human’s drawing into text, when the “drawing” is created by one LSTM, we can get a second LSTM to transform it back into text!

这就是编码器-解码器架构的要点：一个 LSTM 用于编码文本，并与另一个 LSTM 连接以进行解码。正如上一章所讨论的，这不仅限于 LSTM。任何处理 seq2seq 任务（例如摘要）的神经架构都可以是候选架构，为了简洁起见，我们在本章中仅提及 LSTM。图 3-3展示了这种方法。

This is the main point of Encoder-Decoder architecture: one LSTM to encode the text, linked with a second LSTM to decode it. As discussed in the previous chapter, this isn’t limited to LSTMs. Any neural architecture that is tackling a seq2seq task such as summarization is a candidate here, and we refer only to LSTMs in this chapter for brevity. Figure 3-3 illustrates the approach.

图 3-3。用于运行文本摘要示例的编码器-解码器架构（输入和输出序列均因空间原因被截断）。

在这个编码器-解码器架构中，一个 LSTM（编码器）接收待摘要的源文本。处理完“cacti”之后，神经网络的隐藏层表示了从输入序列中收集到的所有信息。第二个 LSTM（解码器）随后以编码器隐藏层的最终状态和一个特殊标记（图中标记为 <START>）作为输入进行初始化，以指示它现在应该开始生成输出。

In this Encoder-Decoder architecture, one LSTM (the encoder) consumes the source text to be summarized. The hidden layer of the neural network after processing “cacti” is a representation of all the information gleaned from the input sequence. A second LSTM (the decoder) is then initialized with input of the final state of the hidden layer of the encoder as well as a special token (denoted in the figure as <START>) to indicate that it should now begin to produce output.

接下来，解码器 LSTM 的工作方式与我们在上一章中讨论的一样，但有一个修改：任何时间步的输入都是前一个时间步的隐藏层，以及它自己在前一个时间步的 token 输出，如图 3-3 中的虚线所示。解码器没有其他依据，因此它必须假设其最佳猜测始终是正确的，并将其用作输入以继续生成序列。

Going forward, the decoder LSTM acts just as we discussed in the previous chapter, with one modification: the input at any time step is the hidden layer from the previous time step, and its own token output from the previous time step, as illustrated by the dotted lines in Figure 3-3. The decoder has nothing else to go on, so it has to assume its best guess is always correct, and uses it as input to continue generating the sequence.

一个合理的问题是，解码器何时停止？就像 <START> 标记一样，词汇表中也有一个特殊的 <STOP> 标记，并且解码器从训练示例中学习到，有时对下一个单词的最佳猜测是声明输出序列已完成（因为所有文本输出最终都应该结束）。因此，当解码器运行时，理想情况下，它最终会生成 <STOP> 标记作为其最佳猜测。1^或者，您可以直接指定输出序列的长度（例如，在 10 个标记后停止）。

A reasonable question is, when does the decoder stop? Just like the <START> token, a special <STOP> token is in the vocabulary, and the decoder learns from its training examples that sometimes the best guess for the next word is to declare the output sequence completed (because all text output should eventually come to an end). Therefore, when the decoder runs, it will, ideally, eventually generate the <STOP> token as its best guess.¹ Alternatively, you might specify the length of the output sequence directly (e.g., stop after 10 tokens).

关于使用编码器-解码器架构的考虑

Considerations on the Use of Encoder-Decoder Architecture

本章我们讨论了一个简化的、受 Pictionary 启发的摘要任务。文本摘要通常更实际地应用于大量文本，例如文档甚至文档集合。事实上，编码器-解码器架构可以应用于许多其他任务，只要它们可以定义为序列到序列。示例包括（但不限于）聊天机器人对话、机器翻译和问答。

We have spent this chapter discussing a simplified, Pictionary-inspired summarization task. Text summarization is usually more practically deployed on much greater volumes of text, such as documents or even document collections. In fact, Encoder-Decoder architecture can be applied to plenty of other tasks, as long as they can be defined as sequence-to-sequence. Examples include (but are not limited to) chatbot conversations, machine translation, and question answering.

更进一步说，输入序列和输出序列的信息媒介不必相同。以自动图像字幕任务为例。在这里，编码器-解码器架构将使用卷积神经网络对图像进行编码，²并使用循环神经网络将结果状态解码为生成的单词序列^。3

Going further, the medium of information is not required to be the same for the input sequence as for the output sequence. Consider the task of automatic image captioning. Here, the Encoder-Decoder architecture would use a Convolutional Neural Network for encoding the image,² and a Recurrent Neural Network for decoding the resulting state into a generated sequence of words.³

我们想对示例中使用的文本摘要任务提出一个重要的说明。这种类型的摘要称为抽象文本摘要( ATS )，因为摘要是使用生成模型创建的。现有的另一种文本摘要称为提取文本摘要( ETS )，它直接从原文中选择单词、短语或句子来构建摘要。ETS 使用截然不同的技术，创建文本的中间表示以识别最重要的片段。它不涉及任何自然语言生成。创建的摘要将忠实于原文，但可能非常不符合语法。另一方面，ATS 更有可能构建符合语法的摘要，但不保证其真实性。对于我们的示例文本，一个事实上不正确（并且有点极端）但语法上完全正确的摘要可能是：“有些舌头吃仙人掌中的蜜源植物。”

We would like to make one important comment on the text-summarization task used in our example. This type of summarization is called abstractive text summarization (ATS) because the summary is created by using a generative model. The other type of text summarization that exists is called extractive text summarization (ETS), which directly selects words, phrases, or sentences from the original text to construct the summary. ETS uses quite different techniques, creating an intermediate representation of the text to identify the most important snippets. It does not involve any natural language generation. The created summary will remain faithful to the original text but may be quite ungrammatical. ATS, on the other hand, is far more likely to construct a grammatical summary but provides no guarantee of veracity. A factually incorrect (and somewhat extreme, to make the point) but perfectly grammatical summary of our running example text might be, “Some tongues eat cacti from nectar plants.”

有趣的是，我们可以结合提取式和抽象式摘要的技术来改善结果。其中一种方法是指针生成网络，如果您感兴趣，我们诚邀您阅读相关内容（但请先读完本报告，因为该技术涉及注意力机制，将在下一章介绍）。

Interestingly, it is possible to combine techniques for extractive and abstractive summarization to improve matters. One approach is pointer-generator networks, and we invite you to read up on them if you are interested (but please finish this report first, as that technique involves an attention mechanism, which is introduced in the next chapter).

关键要点

Key Takeaways

现在，我们通过引入编码器-解码器，构建了神经网络的基础。在继续探讨语言建模神经架构的下一组创新之前，我们想总结一下：

We have now built on the basics of neural networks by introducing encoder-decoders. Before we continue with the next set of innovations in neural architecture for language modeling, we would like to summarize:

• 编码器-解码器是一种用于排列两个神经网络的模块化架构。它可以使用 RNN、LSTM 或其任何变体。
• Encoder-Decoder is a modular architecture for arranging two neural networks. It can use RNNs, LSTMs, or any variations thereof.
• 编码器-解码器架构非常适合 seq2seq 任务，例如文本摘要、翻译、问答、图像标题生成等等（再次注意，我们不仅限于文本数据）。
• Encoder-Decoder architecture is great for seq2seq tasks such as text summarization, translation, question answering, image caption generation, and many others (note, again, that we are not limited to only text data).
• 作为 RNN 的一项改进，输入的长度和输出的长度不必与编码器-解码器匹配。
• As one improvement on RNNs, the length of the input and the length of the output do not have to match with encoders-decoders.
• 基本的编码器-解码器架构存在一个瓶颈问题：源文本中的所有信息都需要以某种方式包含在隐藏层（编码器步骤的末尾）的单个状态中。这会显著降低长输入数据的性能。下一章将讨论如何解决这个问题。
• The basic Encoder-Decoder architecture has a bottleneck problem: all the information from the source text needs to somehow be contained in one single state of the hidden layer (the end of the encoder step). This can significantly degrade performance on long input data. The next chapter discusses how to address this problem.

跨语言交流

Communicating Across Languages

想象一下，你正乘坐邮轮前往巴哈马群岛。你以为已经抵达目的地，船长却用英语告诉你，船似乎在百慕大三角区迷路了，你抵达的是一个未知的岛屿。不幸的是，邮轮燃料不足，无法返回航线，船长请你去和岛民谈谈。

Imagine that you are on a cruise to the Bahamas. Upon what you believe to be your arrival, the ship’s captain informs you, in English, that it seems that the ship has gotten lost inside the Bermuda Triangle, and you have instead arrived at an uncharted island. Unfortunately, the cruise vessel does not have enough fuel to return to course, so the captain asks you to go and speak with the islanders.

听到他们的母语，你却无法辨别出任何你熟悉的语言的相似之处。于是你想出了一个妙招：不如用手指或模仿各种物体和动作，你和岛民一起在沙滩上写下相应的单词。对你来说，他们的书面语言看起来就像一堆以不同形状排列的三角形和圆圈，但它们显然有着相同的词汇和结构概念。

Upon hearing their native language, you are unable to recognize similarities of any language you are familiar with. You come up with a clever idea: why not point to, or mime, various objects and actions, and both you and the islanders will write the appropriate words in the sand. To you, their written language looks like a bunch of triangles and circles arranged in different configurations, but it clearly has the same concept of words and structure.

你指着自己，写下你的名字，然后念出来；岛民也会这样做。你指着船，写下“船”，等等。最终，你把他们的语言映射到几个基本的英语单词上，尽管你自己对如何说、发音或表达他们的语言知之甚少，而且在这个过程中很可能发生了一些误解。你开始尝试把“船需要加油才能离开”这句话翻译成他们的语言。经过反复沟通，你最终表达出了你必须给船加油的基本想法，他们也欣然接受了。

You point at yourself, write your name, and say it; an islander does the same. You point at the boat and write “boat,” and so on. Eventually, you have a mapping of their language to several basic English words, though you have little understanding of how to say, pronounce, or produce their language yourself, and some misunderstandings likely occurred during the process. You get to work trying to translate the English sentence, “The boat needs fuel so it can leave,” into their language. After much back and forth, you communicate the basic thought that you must refuel the boat, to which they happily oblige.

当你终于回到家时，你找到了一本他们语言的字典，并查了一下最终让你回家的那句话：“水龙想要开花；它需要食物。” ¹ 这不是你的意思，但它达到了预期的效果。你继续翻阅字典，意识到了几件事：

When you finally arrive back home, you are able to find a dictionary with their language and check what was said to ultimately get you back home: “Water dragon wants to bloom; it needs food.”¹ This is not what you meant, but it had the intended effect. You keep looking through the dictionary and realize a couple of things:

• 你关于物体的表达相当成功。“水龙”翻译成“船”是合理的；因此，“食物”翻译成“燃料”也是合理的，因为从表面上看，龙需要进食。
• You were pretty successful communicating about objects. “Water dragon” makes sense as a translation for “boat”; and as a result, “food” makes sense for “fuel” since, ostensibly, dragons need to eat.
• 你意识到自己犯了一个错误，把“leave”写成了“bloom”。你指的是树上的叶子，却忘记了“leaves”既是“leaf”的复数，又是“to leave”的变位，这只是英语的一个怪癖。岛民们以为你指的是树上的东西会开花。他们可能很疑惑，为什么你把“水龙”说成是开花，却仍然成功地猜出了你的意思。
• You realize your blunder with “leave” turning into “bloom.” You had pointed to leaves on trees, forgetting that it’s only a quirk of English that “leaves” is both the plural of “leaf” and a conjugation of “to leave.” The islanders thought that you were pointing out what things on trees do, which is bloom. They were probably puzzled as to why you referred to your “water dragon” as blooming but still managed to successfully guess at your meaning.
• 你是分段翻译句子的。你并没有试图一次性传达完整的表述。你从主题——船——以及你想传达的两件事开始：它想离开，以及它需要食物。你一开始似乎坚持认为你的船需要开花，这引起了混乱。但当你继续解释它需要食物时，你和岛民们都能够把这一点放在一边，只关注双方都认同的事实：有一艘船，以及它最后饿了。在翻译句子的后半部分时，你们都只注意（预示！）了句子前半部分的“船”部分。
• You had translated your sentence in chunks. You did not try to get across the entire representation at once. You started with the main topic—the boat—and the two things you wanted to communicate about it: it wanted to leave, and it needed food. You started causing confusion by apparently insisting that your boat needed to bloom. But by the time you moved on to explaining that it wanted food, both you and the islanders were able to set that aside and pay attention to only the agreed-upon fact that there was a boat, and, finally, that it was hungry. In translating the second half of your sentence, you were both paying attention (foreshadowing!) to only the “boat” part of the first half of your sentence.

图 4-1非常简单地说明了这一点。当将“船需要燃料”翻译成“水龙需要食物”时，很容易意识到哪些词是对齐的。

Figure 4-1 illustrates this point extremely simply. When translating “The boat needs fuel” to “Water dragon needs food,” it is easy to realize which words are aligned for the translation.

图 4-1示例翻译部分单词的简单对齐。

总而言之，你与岛民的翻译经历有几个值得注意的方面。首先，你每次只专注于翻译文本的一部分，以免无关信息干扰沟通渠道。其次，你使用了一种合理的翻译变体：传达“燃料”最简单的方式是模仿自己吃东西的动作，这是所有生物都会做的事情，所以对于翻译来说，谁在吃东西并不重要，是你还是船。最后，你使用了另一种变体，这种变体只在一种语言中说得通，但在另一种语言中却变得混乱（“离开”的双重含义）；尽管如此，最终的整体翻译还是成功的。

To summarize, your translation encounter with the islanders had a few notable aspects. First, you focused on only a subset of the text to be translated at a time, so as not to muddle the communication channel with irrelevant information. Second, you used a variation in translation that makes sense: the easiest way to communicate “fuel” was to mime yourself eating, something all creatures do, and so for the translation it did not even matter who was doing the eating, you or the boat. Finally, you used another variation that made sense in only one language but became muddled in the other (the double meaning of “leave”); still, the overall final translation managed to be successful enough.

语言建模的注意力机制：仅关注相关输入

Attention for Language Modeling: Concentrating on Only the Relevant Input

现在，让我们从更专业的术语来讨论如何实现注意力机制，特别是从 Transformer 架构的角度来探讨。注意力机制的动机，同样是为了模仿人类的行为，以及我们在处理特定词语时只关注相关语境的能力。如何在神经网络中实现这一点呢？让我们回顾上一章的编码器-解码器架构。

Let’s now discuss in more technical terms how attention is achieved, specifically through the lens of the Transformer architecture. The motivation for attention is, once again, in the desire to mimic human behavior and our ability to focus on only the relevant context when processing any particular word. How can this be implemented in a neural network? Let’s go back to our Encoder-Decoder architecture in the preceding chapter.

简而言之，注意力机制是神经网络中的另一层，它允许解码器同时参考编码器产生的最终隐藏层以及编码器原始输入的选定子集。换句话说，注意力机制是相关上下文的过滤器。在训练网络其他部分的同时，注意力层也会被训练来决定哪些内容是相关的。

Simply put, an attention mechanism is another layer in the neural network that allows the decoder to consult with both the final hidden layer, produced by the encoder, and a select subset of the original input to the encoder. In other words, attention is a filter for the relevant context. The attention layer is trained to decide what is relevant at the same time as the rest of the network is being trained.

图 4-2突出显示了与图 3-3中的编码器-解码器体系结构的主要体系结构差异。解码器仍然获取编码器隐藏层的最后状态，但现在增加了来自注意层的输入，这指示解码器在每个时间步骤中查看的位置。当编码器完成时，隐藏层保存了有关船、燃料以及前者对后者的需求的信息。但是要翻译“船”，考虑句子中的任何其他内容都是不相关的，因此这里的注意层会将焦点完全放在输入“船”上，并屏蔽其余输入。图 4-2中的示例非常短，但不难想象，当输入文本很长且很复杂时，这将变得多么有用。

Figure 4-2 highlights the main architectural differences from the Encoder-Decoder architecture in Figure 3-3. The decoder still gets the last state of the hidden layer of the encoder, but it is now augmented with input from the attention layer, which directs the decoder where to look at each of its time steps. By the time the encoder is finished, the hidden layer holds information about the boat, fuel, and the former’s need of the latter. But to translate “boat,” considering anything else in the sentence is not relevant, and so the attention layer here would bring the focus entirely on the input, “boat,” and block out the rest of the input. The example in Figure 4-2 is extremely short, but it is easy to imagine how useful this becomes when the input text is long and convoluted.

图 4-2.向编码器-解码器架构添加注意层。

用于语言建模的 Transformer 架构

Transformer Architecture for Language Modeling

在编码器-解码器架构中添加注意力层无疑会有所改进。但从英语的角度来看，我们仍然面临一个奇怪的限制。没错，我们试图从一个序列转换到另一个序列，但为什么这些序列需要从左到右处理呢？毕竟，如果我们的例句是“船要离开，所以需要燃料”呢？对于翻译的最终含义而言，先讨论离开还是先讨论燃料真的很重要吗？

Adding an attention layer to the Encoder-Decoder architecture definitely improves things. But we still have a strange—from our English-language perspective—constraint. Yes, we are trying to go from one sequence to another, but why do these sequences need to be processed left to right? After all, what if our example sentence was, “The boat wants to leave and so it needs fuel”? Does it really matter very much, for the final meaning of the translation, whether the leaving or the fuel is discussed first?

在考虑某个特定词语的上下文时，我们希望能够理解整个句子。事实上，我们常常希望超越单个句子。如果我们的示例包含两个句子——“船需要燃料。它想离开。”——我们就必须跨越句子界限才能看出“它”指的是“船”。

We would like to have access to the whole sentence when considering the context for a particular word. In fact, we often want to go beyond a single sentence. If our running example were two sentences—“The boat needs fuel. It wants to leave.”—we would have to cross sentence boundaries to see that “It” refers to “The boat.”

然而，回想一下，我们最初介绍神经网络时，就指出了使用词窗口的问题，因为词窗口会抓取一定数量词元内的所有上下文。那么，我们如何才能既保留序列到序列的处理过程，又能兼顾处理长序列和注意力层的明显优势，同时又能摆脱固定宽度窗口的限制以及 RNN 架构的强制单向处理呢？

Yet, recall that when we first introduced neural networks, we pointed out the issues with using word windows, which grab all of the context within a certain number of tokens away. How can we still keep a sequence-to-sequence process, with the clear benefits of working with both long sequences and an attention layer, while breaking free from both the restrictions of fixed-width windows and the forced unidirectional processing of an RNN architecture?

Transformer 解决了这些问题。下面是 Transformer 架构的大致轮廓（图 4-3）：首先，堆叠多个编码器层，其中每一层都是神经网络学习关注自身的实现，通过将关注层反馈到其自己的层（而不是像图 4-2中那样反馈到解码器层）来实现。编码器堆叠之后是一堆解码器层，它们执行相同的操作。更准确地说，每对解码器层之间都有两种注意机制，一种用于关注自身，一种用于关注编码器输出。该模型可以访问完整的输入文本，并且输入和输出序列的长度仍然可以不同。

Transformers address these concerns. Here is a broad outline of the Transformer architecture (Figure 4-3): first, stack multiple encoder layers, where each layer is an implementation of the neural network learning to pay attention to itself, by feeding the attention layer back into its own layers (rather than into a decoder layer as in Figure 4-2). The encoder stack is then followed by a stack of decoder layers doing the same thing. To be a little more precise, two attention mechanisms are between every pair of decoder layers, one for paying attention to itself, and one for paying attention to the encoder output. The model has access to the full input text, and the input and output sequences can still be different lengths.

注意力机制现在可以变得更加巧妙。首先，自注意力机制使模型能够理解特定序列中哪些信息是相关的——无论其顺序如何。其次，我们可以采用多种机制来通知隐藏层，而不是使用一种机制（这称为多头注意力机制）。这种方法试图模拟这样一个事实：一个单词作为另一个单词的上下文出现的原因有很多，远远超出了简单的对齐。

The attention mechanism can now get even trickier. First, self-attention allows the model to understand what information is relevant in specific sequences—no matter their order. Second, instead of one mechanism informing a hidden layer, we can have several (this is called multihead attention). This approach is trying to mimic the fact that there are many reasons for a word to appear as context for another word that go far beyond simple alignment.

与我们之前讨论过的架构相比，Transformer 架构有三个关键方面：

The Transformer architecture has three key aspects, in contrast to the architectures we have discussed previously:

神经网络的堆叠层

Stacked layers of neural networks

这本身并非一个新概念，你可能还记得报告前面关于堆叠 RNN 以提高性能的简短评论。但确切的堆叠方式有所不同。我们不再像饼干那样简单地将隐藏层堆叠在一起；现在我们做的是三明治。

This is not a new concept by itself, as you may recall the brief comment on stacking RNNs to improve performance earlier in the report. But precisely how the layers are stacked is different. We are no longer just stacking hidden layers on top of each other like a sleeve of crackers; now we’re making sandwiches.

自注意力层

Self-attention layer

现在，每个编码器都包含一个自注意力层和一个隐藏层，隐藏层是一种注意力机制，用于将当前隐藏层聚焦于前一层的子集。每个解码器都有一个隐藏层和另外两个组件——一个用于自注意力，另一个用于关注编码器。参见图 4-3的简单可视化。

Every encoder now comprises a layer of self-attention as well as a hidden layer, which is an attention mechanism for focusing the current hidden layer on subsets of the previous layer. Every decoder has a hidden layer and two other components—one for self-attention, and the second for paying attention to the encoder. See Figure 4-3 for a simple visualization.

多头自注意力机制

Multihead self-attention

现在有多种方法可以关注词语上下文。自注意力机制现在是多头的。

There is now more than one way to focus on word context. Self-attention is now multiheaded.

图 4-3。Transformer架构。

多头注意力机制

Multihead Attention

为了用一个具体的例子来解释第三点，多头自注意力，我们假设正在翻译“我踢了球”这句话。看一下图 4-4，它展示了我们如何看待注意力信息从一层传播到另一层。

To explain the third point, multihead self-attention, with a concrete example, let’s pretend to be translating the sentence “I kicked the ball.” Take a look at Figure 4-4, which shows how we might think of attention information traveling from one layer to another.

想象一下三种注意力机制并行工作：红色注意力机制关注动作，绿色注意力机制关注谁执行了动作，蓝色注意力机制关注动作的对象是谁（或什么）。那么，对于顶层的“kicked”（踢），红色注意力机制关注与底层相同单词的直接对齐；绿色注意力机制关注“I”（我），蓝色注意力机制关注“ball”（球），三种机制都不太关心冠词“the”（这）。5 在另一个句子^“ She opened the window”（她打开了窗户）中，红色注意力机制关注“opened”（打开），绿色注意力机制关注“she”（她），蓝色注意力机制关注“window”（窗户）。

Imagine three attention mechanisms working in parallel: the red one focuses on actions, the green one focuses on who does the action, and the blue one focuses on who (or what) the action is done to. Then, for “kicked” in the top layer, red attention focuses on the direct alignment with the same word from the bottom layer; green attention focuses on “I,” blue attention focuses on “ball,” and none of them cares much about the article, “the.”⁵ In another sentence, “She opened the window,” red attention focuses on “opened,” green attention focuses on “she,” and blue attention focuses on “window.”

图 4-4。多头注意力机制（来源：Ashish Vaswani 和 Anna Huang 的《生成模型的自注意力》 ）。

让我们立即回顾上一段。例如，我们实际上不能说绿色注意力机制正在寻找句子中“谁”正在做任何特定动作的语义。同样，语言模型不理解含义，也没有语义概念。这对我们来说是一个有用的简写，可以表示一个足够大的语言模型如何发现与已知语言结构紧密相关的关联。

Let’s immediately walk back the preceding paragraph. We cannot actually state that, for example, green attention is looking for the semantics of “who” is doing any particular action in a sentence. Again, LMs don’t understand meaning and have no concepts of semantics. This is helpful shorthand for us to represent the way in which a sufficiently large language model will discover associations that align closely with known linguistic structures.

笔记

原始论文介绍了 Transformer 架构，堆叠了 6 个编码器和 6 个解码器，形成了一个 12 层的深度神经网络^。6

The original paper introducing the Transformer architecture stacked 6 encoders and 6 decoders, to form a deep neural network of 12 layers.⁶

关于使用 Transformer 架构的考虑

Considerations on the Use of Transformer Architecture

我们已经以机器翻译和编码器-解码器架构为例解释了 Transformer 架构，事情已经变得相当复杂了。但我们只差一步：我们是如何从处理 seq2seq 任务的模型，发展到目前最先进的水平的：一个预训练的语言模型（你可能听说过 BERT，即 Transformer 的双向编码器表示），并且可以通过微调（使用更多训练数据进行微调）来处理各种其他任务的？

We’ve explained the Transformer architecture by using the example of machine translation and Encoder-Decoder architecture, and things have already gotten fairly complicated. But we have just one more step to go: how is it that we have also been able to shift from a model that handles seq2seq tasks to the current state of the art: a pretrained language model (you are likely to have heard of BERT, Bidirectional Encoder Representations from Transformers) that can be fine-tuned (tweaked a bit with more training data) to all sorts of other tasks?

Transformer 架构中还有哪些部分在执行“预测下一个单词”的功能？解码器。那么，如果我们舍弃一半的结构，创建一个只包含解码器的架构，会发生什么呢？ OpenAI 推出的生成式预训练 Transformer (GPT) 模型在文本生成方面取得了越来越令人瞩目的成果。但现在我们又回到了单向模型！如何训练一个基于 Transformer 的模型，使其能够同时关注单词的左右上下文？

What part of the Transformer architecture is still playing “predict the next word”? The decoder. So what happens if we throw out half of the structure and create an architecture of only decoders? Generative Pretrained Transformer (GPT) models, introduced by OpenAI, are providing increasingly interesting results in text generation. But now we’ve gone back to only a unidirectional model! How can we train a Transformer-based model that looks at the full context of a word, both to the left and to the right?

拿好我的啤酒，BERT 说^。7

Hold my beer, says BERT.⁷

在训练编码器时，BERT 不会尝试预测下一个单词，而是会屏蔽（或隐藏）输入文本中的一小部分，并尝试进行猜测。BERT 就像在玩填空游戏。现在，该模型可以在猜测时观察每个单词的左右两侧相当远的地方，并重复这个游戏数百万次。完成这个（从各个方面来说都是大规模的）训练过程并构建复杂的语言表征后，该模型就可以用于其他任务了。

Instead of trying to predict the next word while training its encoders, BERT masks, or hides, a small subset of the input text from itself and tries to guess it. BERT is playing fill in the blank. Now the model can look quite far to the left and right of each word as it tries to guess, and it repeats this game millions and millions of times. Having completed this (large-scale, in every sense) training process and built a complex representation of language, the model is now ready to be put to work on other tasks.

我们将简要概述微调如何在一个示例任务中发挥作用：句子分类。此微调步骤的训练数据与传统机器学习分类器的数据类似，由句子和标签对组成。输入序列由一个名为 <CLASS> 的特殊标记构成，后面跟着⁸和句子，输出是标签。BERT 的任务是猜测 <CLASS> 的值。

We will give a brief overview of how fine-tuning works on one example task: sentence classification. The training data for this fine-tuning step looks just like data for traditional machine learning classifiers, comprising pairs of sentences and labels. The input sequence is constructed from a special token called <CLASS>,⁸ followed by the sentence, and the output is the label. BERT is tasked with guessing the value of <CLASS>.

令人惊奇的是，当在多个任务中使用完全相同的预训练 BERT 模型时，这种方法非常有效。此外，这种方法通常比之前专门为执行这些任务而构建和训练的（结构不同的）模型更成功，同时所需的标记训练数据（用于微调）也少得多。这种 Transformer 架构学习到的语言表征既实用又灵活。它基本上将英语知识从原来的填空训练任务迁移到了新的任务（句子分类或其他任务）。这被称为迁移学习。

The amazing thing is that this approach is quite effective when using the exact same pretrained BERT model across multiple tasks. Furthermore, this approach is frequently more successful than previous (differently structured) models that were constructed and trained to perform specifically on those tasks—while needing significantly less labeled training data (for fine-tuning). The language representation that this Transformer architecture has learned is both useful and flexible. It has transferred its knowledge of English generally from its original fill-in-the-blank training task to the new task (sentence classification or other). This is referred to as transfer learning.

终于，我们回到了第一个技术主题：词向量。预训练的 BERT 架构模型可以像 GloVe 或 ELMo 一样用于创建语境化的词向量，并且您可以将这些词向量输入到您的模型中！⁹这是另一种利用 BERT 的方法，而不是直接将其用作模型并根据您的任务进行微调。

Finally, we have come full circle to our first technical topic: embeddings. A pretrained BERT architecture model can be used just like GloVe or ELMo to create contextualized word embeddings, and you can feed these embeddings into your model!⁹ This is another way to make use of BERT, in contrast to using it as a model directly and fine-tuning it to your task.

我们已经达到了语言模型（LM）的当前水平，它们规模庞大、功能强大且资源匮乏。在本报告的下一章（也是最后一章）中，在回顾关键要点之后，我们将深入探讨这种现状的影响。

We have arrived at the current state of the art in LMs, and they are large, powerful, and resource hungry. In the next (and last) chapter of this report, following our review of key takeaways, we get into the implications of this state of affairs.

关键要点

Key Takeaways

本章介绍了语言建模神经网络架构的最新进展。最后，我们再次强调关于注意力机制和 Transformer 架构的一些重要要点：

This chapter has introduced the most recent advancements in neural network architectures for language modeling. We conclude by reemphasizing some important takeaways around the attention mechanism and Transformer architecture:

• 注意力通常通过充当输入文本相关上下文的过滤器来提高序列到序列任务的性能。
• Attention usually improves performance on sequence-to-sequence tasks by acting as a filter for only the relevant context for the input text.
• 注意力机制已经成为 LM 中预期的组成部分。
• Attention mechanisms have become an expected component in LMs.
• 注意力机制似乎为深度学习模型缺乏可解释性提供了一个便捷的答案。然而，正如深度学习的大多数方面一样，事情并非如此简单。我们必须牢记，一切仍然是一种关联，而非推理。
• Attention may seem to provide a convenient answer to the lack of interpretability of deep learning models. Alas, as with most aspects of deep learning, things are just not that simple. We must be careful to remember that everything is still an association, not reasoning.

• Transformer 架构有三个关键方面：

  • 多层编码器和解码器堆叠在一起，每个解码器都单独关注编码器堆栈的输出。

  • 编码器堆栈中的每一对数据与解码器堆栈中的每一层数据之间都夹着一个自注意力层。自注意力层使模型能够理解特定序列中哪些信息是相关的，无论其顺序如何。

  • 在这些自注意力层中，多头自注意力提供了并行的注意力机制，以不同的方式关注词语上下文。

• There are three key aspects to Transformer architecture:

  • Multiple layers of encoders and decoders are stacked, with each decoder paying individual attention to the output from the encoder stack.

  • A self-attention layer is sandwiched between each pair in the encoder stack and each layer in the decoder stack. Self-attention allows the model to understand what information is relevant in specific sequences, no matter the order.

  • Inside these self-attention layers, multihead self-attention provides parallel attention mechanisms for focusing on word context in different ways.

• BERT 是一个强大的基于 Transformer 的语言模型，它并不是进行“预测下一个单词”的训练，而是“填空”。
• BERT is a powerful Transformer-based language model that doesn’t play “predict the next word” to train; it plays “fill in the blank.”
• 迁移学习是指将单个预训练模型（例如 BERT）通过微调应用于多种 NLP 任务（例如分类、问答、翻译和摘要）。这种方法的性能优于其他专门针对这些任务构建的模型，同时所需的标记数据量显著减少（因为 BERT 完全基于未标记数据进行预训练，因此只有微调步骤才需要标记数据）。
• Transfer learning is the use of a single pretrained model, such as BERT, on many NLP tasks (such as classification, question answering, translation, and summarization) by fine-tuning. This approach can outperform other models constructed to perform specifically on those tasks, while needing a significantly smaller amount of labeled data (as BERT is pretrained entirely on unlabeled data, only the fine-tuning step requires labeled data).

语言模型理解什么？

What Do Language Models Understand?

对于人类来说，理解语言或文本的过程通常分为三个部分：处理文本、理解其含义以及与现有知识整合。语言模型（LM），即使是非常复杂的模型，也擅长第一部分，有时还会模仿在第二部分和第三部分观察到的一些行为，尽管它们无法有意识地推理文本的含义。

For humans, the process of understanding language, or text, generally has three parts: process the text, understand its meaning, and integrate with existing knowledge. LMs, even the complex ones, are good at the first part and sometimes mimic some of the behavior observed during the second and third parts, though without being able to intentionally reason about meaning.

让我们举一个具体的例子，回到预测下一个单词的生成任务。我们在网上找到了一个 GPT 模型的演示。我们尝试了^以下简短的实验：

Let’s take a concrete example, returning to the generative task of predicting the next word. We found an available demo online of a GPT model.² We tried the following short experiment:

让我们从最酷的部分开始。GPT 语言模型通过列出城市完成了一个不完整的句子；列出的城市罗马确实是意大利的一个城市；罗马与意大利的关系就像巴黎与法国的关系一样（它们都是首都）。所有这些都无需任何推理或外部知识！接下来，输出中的其余句子都与城市相关，包括食物、天气和文化。该模型没有输出诸如“我喜欢这两个城市的卧室”之类的内容。该模型只列出了与城市相关的方面。当然，其余句子都是完整的、语法正确的，并且与提示一样保持了第一人称。

Let’s start with the cool part. The GPT language model completed an incomplete sentence by listing a city; the city listed, Rome, is indeed a city in Italy; and Rome has the same relationship to Italy as Paris does to France (they are both capitals). All this with no reasoning or external knowledge! Next, the rest of the sentences in the output are all sentences that could reasonably be things to say about cities, which have food, weather, and culture. The model did not output something such as, “I like the bedroom in both of them.” The model listed only aspects that make sense for cities. And, of course, the rest of the sentences are complete, grammatically correct, and remain in the first person, as the prompt did.

另一方面，输出结果并不十分有趣，而且很快就开始重复。在“最喜欢的城市”这个话题上，模型除了单调地列出两个城市或两个城市中每个城市都喜欢的方面之外，没有任何有趣的进展。我们可以假设各种各样的情况：这个提示太短了，或者，由于提示中的句子很短，模型也倾向于使用短句作为输出。但我们无法弄清楚数百万甚至数十亿个参数所形成的关联究竟发生了什么。

On the other hand, the output is not very stimulating and quickly starts to repeat itself. The model does not go anywhere interesting with this topic of favorite cities beyond monotonously listing aspects that are liked in each of the two of them or in both of them. We might hypothesize all sorts of things: that this prompt was too short, or, because the sentences in the prompt were short, the model took a cue to also prefer short sentences for the output. But we can’t figure out what happened with the associations formed by the millions—or billions—of parameters.

我们或许会想，既然模型正确地将罗马识别为与巴黎平行的城市，它就能够正确处理类比。好的，我们试试看。

We may want to think that, since the model correctly identified Rome as the parallel to Paris, it would be able to handle analogies correctly. OK, let’s try.

罗马和意大利的关系在开头再次被正确地呈现。接下来的关系可能还好，也可能不好：纽约不是美国的首都，但绝对是美国最大、最具文化意义的城市之一（就像巴黎和罗马一样）。但进一步说，文本变得非常迷幻，先是说国家有首都是老生常谈（啊？），然后又猜测如果一个国家是另一个国家的首都会发生什么（这不可能），最后还出现了一些带有历史意味的胡言乱语。而且，即使句子的长度和类型各不相同，语法也完全正确。

Once again, the Rome-Italy relationship comes up correctly in the beginning. The next relationship may or may not be OK: New York is not the capital of the United States but is definitely one of the largest and most culturally significant cities (like Paris and Rome). But going further, the text gets really trippy, first calling countries having capitals a truism (huh?), wondering what would happen if one country were the capital of another (not possible), and winding up with some absolute nonsense with a historical flavor. And it’s all wonderfully grammatically correct, even with sentences of different lengths and types.

还有一点需要注意：如果您尝试使用与我们相同的提示运行相同的模型，您可能会得到不同的生成输出。3^有些输出看起来质量更好，有些则更差，我们强烈建议您尝试任何可用的演示，直到您获得足够的轶事数据点来满足您自己的直觉。但对于模型来说，就像对于商场里的孩子一样，这一切都只是“有什么区别？”的问题。就其本身而言，一切都同样有意义。

One more note: if you were to try to run the same model with the same prompts as we did, you would likely get different generated output.³ Some outputs will appear to be of better quality to you, and some of worse, and we strongly encourage you to play around with any available demos until you have enough anecdotal data points to satisfy your own intuition. But to the model, as to the child in the mall, it will all be a case of “what’s the difference?” As far as it’s concerned, it all makes the same amount of sense.

语言模型并不总是足够的

Language Models Are Not Always Enough

到现在为止，希望你已经对语言模型擅长生成易读文本这一事实感到满意了——但它可能并非完全符合事实，句子的组合也可能显得毫无意义。语言模型本身不进行任何推理，也不具备任何常识或任何它们在训练数据中未曾遇到过的知识。因此，一个强大的端到端系统可能会尝试为语言模型添加其他组件，这些组件可以执行某种推理或整合额外的知识（或两者兼而有之）。

By now you’re hopefully quite comfortable with the fact that LMs are good at producing text that reads well—but it absolutely may not be factually correct, and the combination of sentences may appear to be nonsense. LMs do not perform any sort of reasoning and do not by themselves have any idea of common sense or anything that they have not encountered in their training data. Therefore, a robust end-to-end system may seek to augment LMs with other components that can either perform some sort of reasoning or incorporate additional knowledge (or both).

让我们以一个用于制定旅行计划的对话系统（聊天机器人）为例。在对话的每次迭代中，系统采取的步骤大致如下：处理来自人类用户的输入，识别文本的突出部分，收集信息以形成响应（包括对话早期的信息），并以结构良好的自然语言形式返回答案。语言模型 (LM) 擅长处理第一步和最后一步，并且可以为中间步骤提供有用的信息。但是，当完全依靠它们自己时，它们必须尽可能地处理中间步骤，完全依赖于它们在训练期间形成的联想，这可能相当神秘且不一致，尤其是对于像对话这样复杂的任务而言。

Let’s consider a conversation system (a chatbot) for making travel plans as an example. For every iteration in the conversation, the steps the system takes are roughly as follows: process the input from the human user, identify the salient parts of the text, gather the information to form the response (including from earlier in the conversation), and return an answer in the form of well-structured natural language. LMs are great at the first and last steps and can give helpful information for the middle steps. But when left entirely to their own devices, they must fudge the middle steps as best they can, going entirely off associations they have formed during their training, which may be fairly mysterious and inconsistent, especially for a complex task like conversation.

可以添加到系统中的是与语言模型 (LM) 协同执行推理或咨询外部知识的组件；请注意，推理和知识的类型是否必要取决于您的用例。有许多有趣的变体，因此我们仅简要介绍两种，并鼓励您进一步探索。

What can be added to the system are components that work with the LM to perform reasoning or consult with outside knowledge; note that the type of reasoning and knowledge that is, or is not, deemed to be necessary depends on your use case. Many interesting variations are possible, so we will just briefly mention two and encourage you to look into more.

^一种是实体链接的方法。4如果我们拥有一个包含许多已知实体和关系的大型知识库（有时确实如此，具体取决于任务领域），并且结构良好且完全易于理解，那么识别文本中这些实体和关系的存在将大有帮助。以用户话语“我想去意大利。有飞往首都的航班吗？”为例，实体链接过程将识别实体“意大利”和关系“首都”。然后，推理跳跃将在知识库中查询三元组（？，“首都”，“意大利”）中缺失的组成部分并得到结果（“罗马”），从而确定所需的目的地。

One is the approach of entity linking.⁴ If we have a large knowledge base with many known entities and relationships (and we sometimes do, depending on the task domain), which is well structured and fully understandable, identifying the presence of those entities and relationships in the text can go a long way. Taking the sample user utterance, “I’d like to go to Italy. Are there any flights to the capital?,” the process of entity linking would identify the entity “Italy” and the relationship “capitalOf.” A reasoning leap would then query the knowledge base for the missing component of the triple (?, “capitalOf,” “Italy”) and get the result (“Rome”), thus determining the desired destination.

另一种有趣的方法是创建一个新的实体嵌入空间，供语言模型参考。这里的动机是命名实体解析。考虑三个指代同一城市的短语：“NYC”、“纽约市”和“纽约”（最后一个当然在没有上下文的情况下是模棱两可的）。传统的语言模型必然会学习到每个短语略有不同的表示。预训练的实体嵌入模型可以将它们表示为完全相同，以近似地表示它们指的是同一实体。5^在与旅行聊天机器人的对话中，如果人类用户首先提到“从纽约出发旅行”，然后询问“从纽约出发的航班”，这可能有助于系统识别它们指的是同一实体。

Another interesting approach is to create a new embedding space of entities that the language model can consult. The motivation here is named entity resolution. Consider three phrases that all refer to the same city: “NYC,” “New York City,” and “New York” (the last one is, of course, ambiguous without context). A traditional language model will necessarily have learned somewhat different representations of each. A pretrained entity-embedding model could represent these as exactly identical, to approximate the concept that they refer to the same entity.⁵ In the conversation with the travel chatbot, if the human user first mentions “traveling out of NYC” and later asks about “flights from New York,” this may help the system recognize them as referring to the same entity.

重申一下：对于人类而言，理解文本（书面或口头）需要处理文本、理解其含义并与知识进行整合。语言模型完成了第一步，而一个擅长此道的模型（例如 BERT）是一笔真正的财富，因为它只需构建一次，然后用少得多的数据进行微调，就能成功部署到各种任务中，从问答到翻译再到文本生成。从 BERT 的角度来看，它执行的是文本处理的相同步骤。但是语言模型无法执行推理和知识整合的第二步或第三步。尽管它们有时可以做出惊人的模仿，但它们无法区分正确和错误的信息。

To reiterate: for humans, text (written or spoken) comprehension requires processing text, understanding its meaning, and integrating with knowledge. Language models do the first step, and a model that is great at it (such as BERT) is a real asset, as it can be constructed once and fine-tuned with a whole lot less data for successful deployment on all sorts of tasks, from question answering to translation to text generation. From the perspective of BERT, it’s doing the same step of text processing. But LMs cannot perform the second or third steps of reasoning and knowledge integration. Even though they can sometimes give a startling imitation of doing so, they cannot differentiate between correct and incorrect information.

另一方面，对于部署系统的人来说，了解这一点对于所需任务是否重要也很重要。即使没有任何推理或知识，语言模型仍然可以非常有效地完成某些任务。例如，一个经过足够多问答对训练的语言模型，可以在问答任务上表现出色，足以满足实际应用的需求。

On the other hand, it is important for the person deploying the system to understand whether that matters for the required task. LMs can still be quite effective at particular tasks without any reasoning or knowledge. For example, an LM trained on a sufficiently large number of question-and-answer pairs could do an excellent job at the question-answering task and be quite sufficient for the practical application.

能力越大，责任越大

With Great Power Comes Great Responsibility

语言模型 (LM) 目前的发展重点在于规模化和适应性。神经网络的基本设计并非近来才出现，矩阵运算和导数的底层数学知识也远早于我们当前的技术时代。语言模型之所以变得庞大而强大，主要有三个原因：本报告中讨论的深度神经网络架构的发展，尤其是基于 Transformer 的模型；计算能力的并行化提升；以及用于训练这些模型的数据越来越多。

The current story of LMs is one of scale and adaptability. The basic design of neural networks is not recent, and the underlying mathematics of matrix operations and derivatives significantly predate our current technological era. LMs have become large and powerful for three main reasons: the development of the deep neural network architectures discussed in this report, especially Transformer-based models; the parallel improvements in computation power; and, finally, the availability of more and more data for training these models.

因此，我们到达了一个有趣的转折点。我们不再仅仅谈论人们使用相同类型的模型进行机器学习任务（SVM、逻辑回归，甚至 RNN）。现在，我们有越来越多的例子表明，人们使用同一个主要语言模型（例如一个特定的预训练 BERT）并针对各种任务对其进行微调，这既是因为这种方法在各个任务中都取得了成功，也是因为构建和训练大型语言模型需要大量的计算和数据资源。

And so we have arrived at an interesting transition point. We are no longer talking only about people using the same types of models for machine learning tasks (SVMs, logistic regressions, even RNNs). We now have more and more examples of people using the same main language model (such as a specific pretrained BERT) and fine-tuning it for all sorts of tasks, both because this approach has been successful across tasks and because of the really high amount of computing and data resources required to construct and train large LMs.

所有这些经过微调的模型都将继承少数大型 LM 的优点和问题。6^我们将仅讨论其中的一些：

All of these fine-tuned models are then going to inherit both the benefits and the problems of a small number of large LMs.⁶ We will touch on just some of these:

安全

Security

以单一模型为基础可能会成为单点故障，无论是由于数据不良、稳健性失败还是实际攻击造成的。

Having a single model as the basis runs the risk of it becoming a single point of failure, whether as a result of bad data, robustness failure, or actual attacks.

监管

Regulatory

使用 BERT 等模型进行的迁移学习，可以有效地让你在网络上进行训练，然后跨领域、跨行业、跨客户地进行微调和部署。这一过程的法律和隐私问题尚不明确，而且相关法规显然还没有跟上技术的步伐。

The transfer learning that is done with models like BERT effectively allows you to train on the web and then fine-tune and deploy models across domains, industries, and clients. The legal and privacy aspects of this process are far from clear, and the regulations certainly have not yet caught up with the technology.

相信

Trust

大型语言模型的需求重数量轻质量，重不透明轻透明度。正如报告开篇所述，模型擅长生成可信内容，但却无力维护其输出的真实性。仅仅依靠这些模型做出决策可能很危险，因为它们的输出可能难以解释，而且可能会以意想不到的方式失效。此外，由于输出结果可能过于可信，也存在被滥用的风险，例如制造虚假信息，尤其是在针对性内容方面。

The needs of large LMs emphasize quantity over quality and opaqueness over transparency. As discussed in the beginning of the report, models are great at producing plausible content but have no ability to care about the veracity of their output. It may be dangerous to make decisions solely based on these models, whose outputs can be hard to explain, and which can fail in unexpected ways. Furthermore, because the output can be so plausible, opportunities exist for misuse, such as creating disinformation, especially with targeted content.

偏见

Bias

训练数据的内在偏差会逐渐显现。特定的社群和观点不可避免地会被过度或不足地表达，或者与特定的观点相关联（例如，研究人员发现，BERT 会将提及残疾人的短语与更多负面情绪词关联起来）^。7

Intrinsic biases from the training data will make themselves known. Particular communities and perspectives will inevitably be either over- or underrepresented, or associated with particular points of view (as one real example, researchers have found that BERT associates phrases referencing persons with disabilities with more negative sentiment words).⁷

伦理

Ethics

创建像 BERT 这样的大型语言模型所需的规模非常巨大。一方面，训练和运行模型所需的投入可能会对财务和环境造成巨大影响。另一方面，很少有人能够获得足够的资源来做到这一点，而这些开发人员在设计和部署方面的选择可能会产生深远的影响。简而言之，我们应该不断思考“什么时候应该构建模型，什么时候不应该构建模型？”

The scale needed to create a large language model such as BERT is large. On the one hand, the financial and environmental impacts of the necessary efforts to train and run the model can be significant. On the other hand, few people will even have access to the resources to do so, and those developers’ choices in design and deployment may have widespread implications. To put it bluntly, we should continuously be asking “when should a model be built, and when should it not be?”

最后的总结

Final Takeaways

我们希望您已经对语言模型的通用功能有所了解，现在能够放下逻辑细节，与业务中的各利益相关者就高级语言模型概念进行深入的沟通，并且对如何将语言模型应用到您的业务中有足够的信心。最后，我们想分享一些关于人类、机器和语言的想法：

We hope that you have gained an appreciation for the general functionality of LMs, that you will now be able to take a step back from the logistical details and have a thoughtful conversation with the various stakeholders in your business about high-level language model concepts, and that you have reasonable confidence in how to take the next steps toward applying LMs to your business. We would like to leave you with these last few thoughts on humans, machines, and language:

• 当人类交流时，我们不会假设对方的思维方式与自己完全相同。我们假设他们能够解析并理解单词、句子或短语背后的语义信息，并将其抽象化，进而做出回应。
• When humans communicate, we do not assume that the other person is thinking the exact same way to produce a thought. We assume they are able to parse and understand the underlying semantic information of a word, sentence, or phrase, and abstract this and, in turn, respond.
• 人类认为语言既有结构又有含义，而语言模型只感知结构。它们专注于输出语法正确的内容，却不了解语法的存在。它们无法理解词语有意义的概念。同样，语言模型也无法故意撒谎、混淆事实或顾及你的感受。
• While humans interpret language to have both structure and meaning, LMs are aware of only structure. They are entirely focused on outputting something grammatically correct, without the notion that there is such a thing as grammar. They are unable to grasp the concept of words having meaning. In the same way, an LM cannot intentionally tell lies, obfuscate facts, or spare your feelings.
• LM 是对人类互动的增强，而不是替代。
• LMs are an augmentation to human interactions, not a replacement for them.
• LM 实际上并不执行任何推理，但它们经常表现得好像已经执行了推理。
• LMs do not actually perform any reasoning, but they frequently can appear as if they have done so.
• 语言模型 (LM) 是文本处理系统中的组件。LM 并非聊天机器人；它是聊天机器人（或任何应用程序）的文本处理器。在某些用例中，单独使用 LM 可能就足够了。在其他情况下，使用手动规则、外部知识或推理等附加技术可能会更有优势。
• LMs are components in text-processing systems. An LM is not a chatbot; it is the chatbot’s—or any application’s—text processor. For some use cases, using an LM by itself may be sufficient. In other cases, additional techniques for using manual rules, external knowledge, or reasoning may be advantageous.
• LM 是多任务学习者，它们以自己的方式学习大量有关输入文本的信息；然而，最终决定它们用途的是你让它们执行的任务。
• LMs are multitask learners to the extent that they learn, in their way, a ton of information about the input text; however, what ultimately decides their usage is the task you put them to work on.
• 当前最先进的 LM 非常有趣且功能强大，但这些模型越大，我们就越必须仔细考虑其使用的影响和风险。
• Current state-of-the-art LMs are extremely interesting and powerful, but the larger these models become, the more carefully we must consider the implications and risks of their usage.
• 蝙蝠非常酷（见图5-1）。
• Bats are super cool (see Figure 5-1).

图 5-1。一只小棕蝙蝠（来源：美国内政部）。